Quality of Experience in Immersive Video Technologies

Over the last decades, several technological revolutions have impacted the television industry, such as the shifts from black & white to color and from standard to high-definition. Nevertheless, further considerable improvements can still be achieved to provide a better multimedia experience, for example with ultra-high-definition, high dynamic range & wide color gamut, or 3D. These so-called immersive technologies aim at providing better, more realistic, and emotionally stronger experiences. To measure quality of experience (QoE), subjective evaluation is the ultimate means since it relies on a pool of human subjects. However, reliable and meaningful results can only be obtained if experiments are properly designed and conducted following a strict methodology. In this thesis, we build a rigorous framework for subjective evaluation of new types of image and video content. We propose different procedures and analysis tools for measuring QoE in immersive technologies. As immersive technologies capture more information than conventional technologies, they have the ability to provide more details, enhanced depth perception, as well as better color, contrast, and brightness. To measure the impact of immersive technologies on the viewersâ QoE, we apply the proposed framework for designing experiments and analyzing collected subjectsâ ratings. We also analyze eye movements to study human visual attention during immersive content playback. Since immersive content carries more information than conventional content, efficient compression algorithms are needed for storage and transmission using existing infrastructures. To determine the required bandwidth for high-quality transmission of immersive content, we use the proposed framework to conduct meticulous evaluations of recent image and video codecs in the context of immersive technologies. Subjective evaluation is time consuming, expensive, and is not always feasible. Consequently, researchers have developed objective metrics to automatically predict quality. To measure the performance of objective metrics in assessing immersive content quality, we perform several in-depth benchmarks of state-of-the-art and commonly used objective metrics. For this aim, we use ground truth quality scores, which are collected under our subjective evaluation framework. To improve QoE, we propose different systems for stereoscopic and autostereoscopic 3D displays in particular. The proposed systems can help reducing the artifacts generated at the visualization stage, which impact picture quality, depth quality, and visual comfort. To demonstrate the effectiveness of these systems, we use the proposed framework to measure viewersâ preference between these systems and standard 2D & 3D modes. In summary, this thesis tackles the problems of measuring, predicting, and improving QoE in immersive technologies. To address these problems, we build a rigorous framework and we apply it through several in-depth investigations. We put essential concepts of multimedia QoE under this framework. These concepts not only are of fundamental nature, but also have shown their impact in very practical applications. In particular, the JPEG, MPEG, and VCEG standardization bodies have adopted these concepts to select technologies that were proposed for standardization and to validate the resulting standards in terms of compression efficiency

A QUALITY ASSESSMENT PROTOCOL FOR FREE-VIEWPOINT VIDEO SEQUENCES SYNTHESIZED FROM DECOMPRESSED DEPTH DATA

International audienceIn this paper, the analysis of a subjective quality experiment consisting in assessing the quality of free-viewpoint video sequences generated from decompressed depth data is pre- sented. In the absence of a dedicated subjective assessment protocol for the evaluation of such 3D systems, a subjective quality assessment methodology is proposed for the context of MVD compression. The proposed methodology includes the assessment of free-viewpoint video sequences generated from decompressed depth data and from view synthesis pro- cesses. The proposed methodology is meant to assess the per- formances of depth map compression and view synthesis al- gorithms

Subjective evaluation of two stereoscopic imaging systems exploiting visual attention to improve 3D quality of experience

Crosstalk and vergence-accommodation rivalry negatively impact the quality of experience (QoE) provided by stereoscopic displays. However, exploiting visual attention and adapting the 3D rendering process on the fly can reduce these drawbacks. In this paper, we propose and evaluate two different approaches that exploit visual attention to improve 3D QoE on stereoscopic displays: an offline system, which uses a saliency map to predict gaze position, and an online system, which uses a remote eye tracking system to measure real time gaze positions. The gaze points were used in conjunction with the disparity map to extract the disparity of the object-of-interest. Horizontal image translation was performed to bring the fixated object on the screen plane. The user preference between standard 3D mode and the two proposed systems was evaluated through a subjective evaluation. Results show that exploiting visual attention significantly improves image quality and visual comfort, with a slight advantage for real time gaze determination. Depth quality is also improved, but the difference is not significant

Quality Assessment of a Stereo Pair Formed From Two Synthesized Views Using Objective Metrics

When a stereo pair is formed from two synthesized views, it is unclear whether objective 2D quality metrics can provide a good estimation of the perceived quality. In this paper, this problem is addressed considering a 3D video represented in multiview video plus depth format. The performance of different state-of-the-art 2D quality metrics is analyzed in terms of correlation with subjective perception of video quality. A set of subjective data collected through formal subjective evaluation tests is used as benchmark. Results show that some objective metrics, including PSNR, do not predict well perceived quality of synthesized views. On the other hand, metrics such as VIF, VQM, MS-SSIM, or SSIM have a high correlation with perceived quality

On the evaluation of 3D codecs on multiview autostereoscopic display

Evaluating the performance of different 3D codecs on multiview autostereoscopic monitor is a tedious process, as it requires the synthesis of a dense set of views. Therefore, it is legitimate to ask if evaluations could be performed on stereoscopic monitors instead and could lead to similar results as on multiview autostereoscopic monitors. This paper tries to answer to this question by analyzing a set of subjective scores resulting from evaluations of different 3D codecs on both display technologies. Results show that the comparison of different 3D codecs on stereoscopic display leads to similar conclusions when compared to multiview autostereoscopic display

Quality Assessment of a Stereo Pair Formed From Decoded and Synthesized Views Using Objective Metrics

When a stereo pair is formed from a decoded view and a synthesized view, it is unclear how the overall quality of the stereo pair should be assessed through objective quality metrics. In this paper, this problem is addressed considering a 3D video represented in the format of multiview video plus depth. The performance of different state-of-the-art 2D quality metrics is analyzed in terms of correlation with subjective perception of video quality. A set of subjective data collected through formal subjective evaluation tests is used as benchmark. Results show that the measured quality of the decoded view has the highest correlation with perceived quality. If the objective quality assessment is based on the measured quality of the synthesized view, it is suggested to use VIF, VQM, MS-SSIM, or SSIM since they significantly outperform other objective metrics, including PSNR

Calculation of average coding efficiency based on subjective quality scores

The Bjøntegaard model is widely used to calculate the coding efficiency between different codecs. However, this model might not be an accurate predictor of the true coding efficiency as it relies on PSNR measurements. Therefore, in this paper, we propose a model to calculate the average coding efficiency based on subjective quality scores, i.e., mean opinion scores (MOS). We call this approach Subjective Comparison of ENcoders based on fItted Curves (SCENIC). To consider the intrinsic nature of bounded rating scales, a logistic function is used to fit the rate–distortion (R–D) values. The average MOS and bit rate differences are computed between the fitted R–D curves. The statistical property of subjective scores is considered to estimate corresponding confidence intervals on the calculated average MOS and bit rate differences. The proposed model is expected to report more realistic coding efficiency as PSNR is not always correlated with perceived visual quality

Predicting 3D Quality based on Content Analysis

Development of objective quality metrics that can reliably predict perceived quality of 3D video sequences is challenging. Various 3D objective metrics have been proposed, but PSNR is still widely used. Several studies have shown that PSNR is strongly content dependent, but the exact relationship between PSNR values and perceived quality has not been established yet. In this paper, we propose a model to predict the relationship between PSNR values and perceived quality of stereoscopic video sequences based on content analysis. The model was trained and evaluated on a dataset of stereoscopic video sequences with associated ground truth MOS. Results showed that the proposed model achieved high correlation with perceived quality and was quite robust across contents when the training set contained various contents

EyeC3D: 3D video eye tracking dataset

Despite efforts of the scientific community in recent years, little is known about the mechanisms of the human visual system that control visual attention when watching 3D content. To help understanding these mechanisms and develop more accurate visual attention models, we created a public 3D video eye tracking dataset. The dataset provides the eye tracking information corresponding to eight stereoscopic video sequences. The eye tracking information includes the fixation points and fixation density maps measured during subjective experiments. This paper describes the dataset in details, including the stereoscopic video sequences, the eye tracking experiments, and the computation of the fixation density maps

How to Benchmark Objective Quality Metrics from Paired Comparison Data?

The procedures commonly used to evaluate the performance of objective quality metrics rely on ground truth mean opinion scores and associated confidence intervals, which are usually obtained via direct scaling methods. However, indirect scaling methods, such as the paired comparison (PC) method, have a higher discriminatory power and are gaining popularity, for example in crowdsourcing evaluations. In this paper, we present an existing analysis tool, the classification errors, which can also be used for PC data. Additionally, we propose a new analysis tool based on the receiver operating characteristic analysis. This tool can be used to further assess the performance of objective metrics based on PC data. We provide a MATLAB script with an implementation of the proposed tools and we show one example of application of the proposed tools